Method and system for assisting user with product handling in retail shopping

ABSTRACT

The present invention discloses a method and a system for assisting user with product handling in a retail shopping. The method comprising identifying one or more products from a list of products that user intends to purchase, neighbouring products and orientations of the one or more products and the neighbouring products from a plurality of image frames and rack information, comparing the identified orientations with historic orientations of products and neighbouring products associated with the rack information, identifying at least one product from the one or more products and the neighbouring products that is improperly oriented based on comparison, determining product safety handling instructions of the identified at least one product and assisting the user in real-time with handling of the identified at least one product based on the product safety handling instructions.

TECHNICAL FIELD

The present subject matter is generally related to enhancing retailinteraction of a user in real-time, more particularly, but notexclusively, to a method and a system for assisting a user with producthandling in a retail shopping environment.

BACKGROUND

In retail shopping environment, there are many supporting devices andapplications that aim to provide seamless shopping experience to users.Most of these solutions aim at detection of products, selection ofrequired products, guiding the users to a rack based-oninterested/requested products and providing the users with smart cartfor an easy user experience. However, the challenge arises when a personwants to access displayed products in racks of a retail store. Therewould be products or objects on the rack or on a route to the rack,which if not handled correctly would cause harm or injury to the persondue to nature of object or placement of the object. For example, aperson may get injured or harmed if he/she tries to pick up a productlike a sharp object with protruding edges, for example, knife that isnot placed correctly. Furthermore, a person of low height will not beable to reach a specific product and sometimes end up not picking itcorrectly, which may cause harm or injury to the person. For example, akid should be guided in the right way to pick a product that is placedhigh in the rack above his/her height. Most of the time, theabove-mentioned scenarios evolve dynamically and are not addressed bythe existing solutions.

The information disclosed in this background of the disclosure sectionis for enhancement of understanding of the general background of theinvention and should not be taken as an acknowledgement or any form ofsuggestion that this information forms the prior art already known to aperson skilled in the art.

SUMMARY

In an embodiment, the present disclosure may relate to a method ofassisting user with product handling in a retail shopping in real-time.The method includes extracting a list of products from a handheld deviceassociated with a user and historic user details of the user from adatabase, receiving a plurality of image frames of one or more productsfrom the list of products and neighbouring products in a rack from thehandheld device and Internet of Things (IoT) sensors attached to therack, identifying at least one of product type of the one or moreproducts, neighbouring product type of the neighbouring products andorientations of the one or more products and the neighbouring productsfrom the plurality of image frames and rack information corresponding tothe rack stored in the database, comparing the identified orientationsof the one or more products and the neighbouring products with historicorientations of products and neighbouring products associated with therack from the database, identifying at least one product from the one ormore products and the neighbouring products that is improperly orientedbased on comparison, determining product safety handling instructions ofthe identified at least one product based on historic product detailsstored in the database and the identified orientations of the at leastone product, and assisting the user in real-time with handling of theidentified at least one product based on the product safety handlinginstructions.

In an embodiment, the present disclosure may relate to a productassistance system for assisting user with product handling in a retailshopping in real-time. The product assistance system may include aprocessor and a memory communicatively coupled to the processor, whereinthe memory stores processor-executable instructions, which on execution,cause the processor to extract a list of products from a handheld deviceassociated with a user and historic user details of the user from adatabase, receive a plurality of image frames of one or more productsfrom the list of products and neighbouring products in a rack from thehandheld device and IoT sensors attached to the rack, identify at leastone of product type of the one or more products, neighbouring producttype of the neighbouring products and orientations of the one or moreproducts and the neighbouring products from the plurality of imageframes and rack information corresponding to the rack stored in thedatabase, compare the identified orientations of the one or moreproducts and the neighbouring products with historic orientations ofproducts and neighbouring products associated with the rack from thedatabase, identify at least one product from the one or more productsand the neighbouring products that is improperly oriented based oncomparison, determine product safety handling instructions of theidentified at least one product based on historic product details storedin the database and the identified orientations of the at least oneproduct, and assist the user in real-time with handling of theidentified at least one product based on the product safety handlinginstructions.

In an embodiment, the present disclosure may relate to a non-transitorycomputer readable medium including instructions stored thereon that whenprocessed by at least one processor cause a product assistance system toperform operations comprising extracting a list of products from ahandheld device associated with a user and historic user details of theuser from a database, receiving a plurality of image frames of one ormore products from the list of products and neighbouring products in arack from the handheld device and Internet of Things (IoT) sensorsattached to the rack, identifying at least one of product type of theone or more products, neighbouring product type of the neighbouringproducts and orientations of the one or more products and theneighbouring products from the plurality of image frames and rackinformation corresponding to the rack stored in the database, comparingthe identified orientations of the one or more products and theneighbouring products with historic orientations of products andneighbouring products associated with the rack from the database,identifying at least one product from the one or more products and theneighbouring products that is improperly oriented based on comparison,determining product safety handling instructions of the identified atleast one product based on historic product details stored in thedatabase and the identified orientations of the at least one product,and assisting the user in real-time with handling of the identified atleast one product based on the product safety handling instructions.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the drawings and the followingdetailed description.

BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate exemplary embodiments and togetherwith the description, serve to explain the disclosed principles. In thefigures, the left-most digit(s) of a reference number identifies thefigure in which the reference number first appears. The same numbers areused throughout the figures to reference like features and components.Some embodiments of system and/or methods in accordance with embodimentsof the present subject matter are now described below, by way of exampleonly, and with reference to the accompanying figures.

FIG. 1 illustrates an exemplary environment for assisting user withproduct handling in a retail shopping in real-time in accordance withsome embodiments of the present disclosure.

FIG. 2 shows a detailed block diagram of a product assistance system inaccordance with some embodiments of the present disclosure.

FIG. 3 illustrates a flowchart showing a method for populating adatabase to be used in assisting user with product handling inaccordance with some embodiments of present disclosure.

FIG. 4a -FIG. 4b illustrates an exemplary representation of assistinguser with product handling in a retail shopping in real-time inaccordance with some embodiments of present disclosure.

FIG. 5 illustrates a block diagram of an exemplary computer system forimplementing embodiments consistent with the present disclosure.

It should be appreciated by those skilled in the art that any blockdiagrams herein represent conceptual views of illustrative systemsembodying the principles of the present subject matter. Similarly, itwill be appreciated that any flowcharts, flow diagrams, state transitiondiagrams, pseudo code, and the like represent various processes whichmay be substantially represented in computer readable medium andexecuted by a computer or processor, whether or not such computer orprocessor is explicitly shown.

DETAILED DESCRIPTION

In the present document, the word “exemplary” is used herein to mean“serving as an example, instance, or illustration.” Any embodiment orimplementation of the present subject matter described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments.

While the disclosure is susceptible to various modifications andalternative forms, specific embodiment thereof has been shown by way ofexample in the drawings and will be described in detail below. It shouldbe understood, however that it is not intended to limit the disclosureto the particular forms disclosed, but on the contrary, the disclosureis to cover all modifications, equivalents, and alternatives fallingwithin the scope of the disclosure.

The terms “comprises”, “comprising”, or any other variations thereof,are intended to cover a non-exclusive inclusion, such that a setup,device or method that comprises a list of components or steps does notinclude only those components or steps but may include other componentsor steps not expressly listed or inherent to such setup or device ormethod. In other words, one or more elements in a system or apparatusproceeded by “comprises . . . a” does not, without more constraints,preclude the existence of other elements or additional elements in thesystem or method.

In the following detailed description of the embodiments of thedisclosure, reference is made to the accompanying drawings that form apart hereof, and in which are shown by way of illustration specificembodiments in which the disclosure may be practiced. These embodimentsare described in sufficient detail to enable those skilled in the art topractice the disclosure, and it is to be understood that otherembodiments may be utilized and that changes may be made withoutdeparting from the scope of the present disclosure. The followingdescription is, therefore, not to be taken in a limiting sense.

Embodiments of the present disclosure provide a solution for assisting auser for handling harmful product in a retail shopping environment.Furthermore, the present disclosure uses a new mechanism to build aproduct model to identify products, which the user has interest topurchase, its neighbouring products and position of the product and itsneighbouring products. The present disclosure uses Capsule NeuralNetwork (Capsule Net model), which is trained with all the productimages in all orientations, which can classify a given product intodifferent classes. The characteristic of Capsule Net model is that itcan classify the image of the product into correct class given theproduct in any orientation in the image. Later, the images of productsin different orientation are annotated with appropriate safety handlingcaptions depending on the product characteristics and position. Thesecaptions and penultimate layer output of the Capsule Net model and theclassification are given as input to the bidirectional Long Short-TermMemory (LSTM) network, which is trained to generate the guided Englishtext. The guided English text is then fine-tuned as an action orinstruction message in Natural Language Processing (NLP) system withappropriate instructions to the user based on user details and usercharacteristics. This approach, also, guides visually disabled person onhow to handle hazardous, misplaced, disorientated or wrongly placedobjects in the retail shopping environment. In addition, the presentdisclosure guides the user to prevent or avoid any dangerous situationsin the retail shopping environment by making sure safe route is alwayssuggested.

FIG. 1 illustrates an exemplary environment for assisting user withproduct handling in a retail shopping in real-time in accordance withsome embodiments of the present disclosure.

As shown in the FIG. 1, the environment 100 includes a handheld device101, a database 103, a communication network 105 and a productassistance system 107. The handheld device 101 may be connected throughthe communication network 105 to the product assistance system 107. Inan embodiment, the handheld device 101 may include, but is not limitedto, a mobile terminal, a tablet computer, a camera or Personal DigitalAssistant (PDA). A person skilled in the art would understand that, anyelectronic devices with a camera feature, not mentioned explicitly, mayalso be used as the handheld device 101 in the present disclosure. Thehandheld device 101 may provide image frames to the product assistancesystem 107 via the communication network 105 and may communicate withthe product assistance system 107 via the communication network 105 inuser preferred medium to assist a user. The image frames may be at leastone of image and video. The handheld device 101 may extract item/productinformation by focusing on racks that contains one or more products thatthe user may intend to purchase. The handheld device 101 may giveposition of products and details of the products as images from a closerview. The communication network 105 may include, but is not limited to,a direct interconnection, an e-commerce network, a Peer-to-Peer (P2P)network, Local Area Network (LAN), Wide Area Network (WAN), wirelessnetwork (for example, using Wireless Application Protocol), Internet,Wi-Fi, Bluetooth and the like.

In the embodiment, the product assistance system 107 may assist the userwith product handling in a retail shopping in real-time based on atleast one product identified from one or more products and neighbouringproducts and their orientations. The product assistance system 107 mayinclude an I/O interface 111, a memory 113 and a processor 115. The I/Ointerface 111 may be configured to receive the image frames from thehandheld device 101. Analogously, the I/O interface 111 may beconfigured to communicate with the handheld device 101 using the userpreferred medium to assist the user. The I/O interface 111 may employcommunication protocols/methods such as, without limitation, audio,analog, digital, monoaural, Radio Corporation of America (RCA)connector, stereo, IEEE-1394 high speed serial bus, serial bus,Universal Serial Bus (USB), infrared, Personal System/2 (PS/2) port,Bayonet Neill-Concelman (BNC) connector, coaxial, component, composite,Digital Visual Interface (DVI), High-Definition Multimedia Interface(HDMI), Radio Frequency (RF) antennas, S-Video, Video Graphics Array(VGA), IEEE 802.11b/g/n/x, Bluetooth, cellular e.g., Code-DivisionMultiple Access (CDMA), High-Speed Packet Access (HSPA+), Global Systemfor Mobile communications (GSM), Long-Term Evolution (LTE), Worldwideinteroperability for Microwave access (WiMax), or the like.

The image frames received by the I/O interface 111 and a list ofproducts extracted from the handheld device 101 associated with the usermay be stored in the memory 113. The memory 113 may be communicativelycoupled to the processor 115 of the product assistance system 107. Thememory 113 may, also, store processor instructions which may cause theprocessor 115 to execute the instructions for assisting the user withproduct handling. The memory 113 may include, without limitation, memorydrives, removable disc drives, etc. The memory drives may furtherinclude a drum, magnetic disc drive, magneto-optical drive, opticaldrive, Redundant Array of Independent Discs (RAID), solid-state memorydevices, solid-state drives, etc.

The processor 115 may include at least one data processor for assistingthe user with product handling. The processor 115 may includespecialized processing units such as integrated system (bus)controllers, memory management control units, floating point units,graphics processing units, digital signal processing units, etc.

In the embodiment, the product assistance system 107 may exchange datawith a database 103 directly or through the communication network 105.The database 103 may be populated or stored with historic data thatincludes at least one of historic images of products, historic images ofneighbouring products, historic orientations of products andneighbouring products, historic product details of the products and theneighbouring products and historic user details. Here, the historic datamay refer to data received from one or more handheld devices 101 by theproduct assistance system 107 during training phase. The historic userdetails may comprise at least one of user age, user gender, userpreferences, user behavioural characteristics, user disabilitycharacteristics, user height and user weight. The historic productdetails may comprise at least one of product name, product type, productbeing properly oriented or improperly oriented, product handlinginstruction, the rack information and product characteristics. Here, thehistoric orientations of products and neighbouring products refer to allthe possible orientations of product and neighbouring products that canbe envisioned when a product is placed in a rack.

The database 103 may, also, be updated at pre-defined intervals of time.These updates may be related to at least one of the historic images ofproducts, the historic images of neighbouring products, the historicorientations of products and neighbouring products, the historic productdetails of the products and the neighbouring products and thehistoric-user details for adaptive learning.

FIG. 2 shows a detailed block diagram of a product assistance system inaccordance with some embodiments of the present disclosure.

The product assistance system 107, in addition to the I/O interface 111and processor 115 described above, may include data 200 and one or moremodules 211, which are described herein in detail. In the embodiment,the data 200 may be stored within the memory 113. The data 200 mayinclude, for example, product list data 201, image frame data 203 andother data 205.

The product list data 201 may include a list of products that a userintends to purchase in a shop. The product list is extracted by theproduct assistance system 107 from the handheld device 101 associatedwith the user and stored in the product list data 201.

The image frame data 203 may include a plurality of image frames of oneor more products from the list of products that the user intends topurchase and neighbouring products in a rack. The plurality of imageframes are received by the product assistance system 107 from thehandheld device 101 and Internet of Things (IoT) sensors attached to therack and stored in the image frame data 203. Here, the image frames mayinclude at least one of image and video.

The other data 205 may store data, including temporary data andtemporary files, generated by modules 211 for performing the variousfunctions of the product assistance system 107.

In the embodiment, the data 200 in the memory 113 are processed by theone or more modules 211 present within the memory 113 of the productassistance system 107. In the embodiment, the one or more modules 211may be implemented as dedicated hardware units. As used herein, the termmodule refers to an Application Specific Integrated Circuit (ASIC), anelectronic circuit, a Field-Programmable Gate Arrays (FPGA),Programmable System-on-Chip (PSoC), a combinational logic circuit,and/or other suitable components that provide the describedfunctionality. In some implementations, the one or more modules 211 maybe communicatively coupled to the processor 115 for performing one ormore functions of the product assistance system 107. The said modules211 when configured with the functionality defined in the presentdisclosure will result in a novel hardware.

In one implementation, the one or more modules 211 may include, but arenot limited to, an extraction module 213, a receiver module 215, anidentifier module 217, a comparison module 219, a determiner module 221,an assisting module 223, a guided instruction module 225 and a preferredmedium module 227. The one or more modules 211 may, also, include othermodules 229 to perform various miscellaneous functionalities of theproduct assistance system 107.

Here, the receiver module 215 and the other modules 229 may be used in atraining phase. During the training phase, the database 103 may bepopulated or stored with historic data that includes at least one ofhistoric images of products, historic images of neighbouring products,historic orientations of products and neighbouring products, historicproduct details of the products and the neighbouring products andhistoric user details. The historic data may refer to data receivedduring the training phase.

The extraction module 213 may extract a list of products from thehandheld device 101 associated with a user and historic user details ofthe user from the database 103. The extraction module 213 may, also,extract rack information about various displayed product/item detailsand product positional information. This information along with stockavailability may be updated in the database 103 at predefined intervalsof time. For instance, as a user enters a store, the user may present alist of products he/she wants to buy. This list may be extracted fromthe handheld device 101 associated with the user. At the same time,historic user details of the user may be extracted from the database103.

The other modules 229 may provide a shopping route to access the list ofproducts in a shop based on at least one of the list of products, thehistoric user details and product purchase priority of the user. Forinstance, based on the list of products that user intends to purchase orbased on user preference from the historic user details stored in thedatabase 103 a shopping route to access the products may be provided. Incase, the user prioritizes the order in which he/she may want topurchase products, then the user may be provided with a shopping routebased on his/her product purchase priority.

The receiver module 215 may receive a plurality of image frames of oneor more products from the list of products and neighbouring products ina rack from the handheld device and IoT sensors attached to the rack.For instance, as the user walks through the route to buy the products,the user may use the handheld device 101 to capture the information ofthe products he/she wants to buy. The user may move the device 101 tocover an area around the product so that different image frames arecaptured which would help in knowing the type of product, neighbouringproducts and orientation of the products. This information is collectedalong with the image frames collected by the IoT sensors attached to thecorresponding rack. Like the handheld device 101, the IoT sensors, also,capture image frames of the products to be bought by the user. The IoTsensors allow to capture a bigger area of the rack, vicinity and detailsof the products from different directions, which would give more detailson position of the products and neighbouring products in a rack.

The identifier module 217 may identify at least one of product type ofthe one or more products, neighbouring product type of the neighbouringproducts and orientations of the one or more products and theneighbouring products from the plurality of image frames and rackinformation corresponding to the rack stored in the database. The imagesframes obtained from the receiver module 215 are sent to the identifiermodule 217. In the identifier module 217, these images are given to deeplearning object detection model like Region Convolutional Neural Network(RCNN) or Stochastic Gradient Descent (SGD) Neural Network or You OnlyLook Once (YOLO) Neural Network to identity the product/s from the imageframes. If multiple products i.e. neighbouring products are identifiedfrom a single image frame, each of the product and their correspondingimages are stored in the database 103.

The comparison module 219 may compare the orientations of the one ormore products and the neighbouring products that are identified by theidentifier module 217 with historic orientations of products andneighbouring products associated with the rack from the database 103.The comparison is performed to identify if any of the one or moreproducts and the neighbouring products are not oriented properly suchthat it is unsafe to handle that may cause injury or positionedimproperly such that it may fall from the rack.

The identifier module 217 may identify at least one product from the oneor more products and the neighbouring products that is improperlyoriented based on comparison result obtained from the comparison module219. For example, a user is buying a product, which has an oil can nearto the product, which is leaking. If the user tries to touch or pick theproduct that is greasy due to oil leak, he/she may panic and if the oilis further spilled by chance it may cause injury to the user in casehe/she walks over the oil. In such situation, the identifier module 217may identify oil can next to the product that is improperly oriented orunsafe to handle.

The determiner module 221 may determine product safety handlinginstructions of the at least one product identified by the identifiermodule 217 based on historic product details stored in the database 103and the identified orientations of the at least one product. In thisstep, the determiner module 221 may use image pruning techniques for theplurality of image frames of one or more product images and neighbouringproducts that are were collected from the handheld device 101 and IoTsensors to get rid of unwanted images. These images are given as inputto the Capsule Net model for object or product classification. TheCapsule Net model is trained on all the products in the store toclassify them according to their product details. Based on these productdetails and their orientations in the rack, the products are identifiedas either safe or unsafe category. It may also be categorized asproperly oriented or improperly oriented. These results are given to thebidirectional LSTM model, which outputs the appropriate guided Englishtext based on the orientation of the product and product details. Totrain LSTM, the image frames are annotated with text indicating how theproduct in the image has to be handled based on its orientation. Forinstance, suppose an object is protruding along the edge of a rack,which may fall when user tries to pick it up. In this case trying topick that product would make the product fall most of the time. For suchproducts that are on the edge of the rack and may fall when tried topick it up, the image may be annotated as, “Please push the productslightly on the rack to the left and then pick it up.” All unsafeproducts like knife, consumer goods, razors, mechanic products, glassitems, ceramic products, sharp products may be stored hierarchicallywith the most unsafe product on the top. The messages are annotatedaccordingly based on the unsafe product in the hierarchy. For instance,suppose the products like knife, glass item, shoes, biscuit pack are insequence then the corresponding action or instruction messages will beannotated as below:

For knife: “Be extra careful!! The object has sharpened edge towards theleft and blunt side on the right. Gently move it to the back beforelifting it using the blunt side of the object”.

For glass: “Please handle with extra care while you push it back sinceit is brittle object. Hold the entire object in hand while pushing itback”.

For shoes: “Please push the object back on rack, use less force”.

For biscuit pack: “Please push the object back on rack, it is lightweight. Use minimum force”.

These annotated images are given to bidirectional LSTM model and it istrained to generate text (instructions to handle the object/product).The model generates distinct text instructions for same objects but indifferent orientation. Whenever the identifier module 217 identifies theproduct is in unsafe position, the determiner module 221 feeds the imageto the LSTM model and generates the instructions in English text on howto handle the product.

The assisting module 223 may assist the user in real-time with handlingof the at least one product based on the product safety handlinginstructions determined by the determiner module 221.

The guided instruction module 225 may convert the product safetyhandling instructions to customized guided instructions based on thehistoric user details and the historic product details.

The guided instruction module 225 may receive the guided English textfrom the determiner module 221 generated by the LSTM model. The guidedtext is specific to the product and the orientation of the product. Theguided instruction module 225 may take historic user details and thehistoric product details from the database 103 and present it to deeplearning NLP system along with the received guided English text. Theoutput of the NLP system may be the customized guided instructions basedon the user and product details. For example, for adults of all ages >15years, the customized guided instruction may be as given below:

“This is sharp item, it is placed at two racks above. The left edge isprotruded, be careful and use the blunt side of the object which is onthe right side to handle it”.

For kids <15 years or disabled persons who are on wheel chair or adultswhose height is <height of the rack where product is placed, thisproduct will not be allowed to be bought by them since it is unsafe.

For persons who are senior citizens, either the person is not be allowedto buy or the customized guided instruction may be as given below:

“Please be extra careful!!! This is sharp item, it is placed at tworacks above. The left edge is protruded, be careful and use the bluntside of the object which is on the right to handle it”.

If the disabilities characteristics indicates not to allow sharp itemsto be handled, then the product will not be allowed to be bought by thatperson.

Based on the product shape, size and placement in the rack, variousinstructions are generated for handling all the dangerous objectssafely. As explained above based on the product orientation on the rack,how much movement is required or how to approach the product is decided.For dangerous objects like “knife”, customized guided instructions areprovided with exact movement of hand position to approach the blunt endof the knife.

The preferred medium module 227 may convert the customized guidedinstructions into a user preferred medium based on the historic userdetails. The customized guided instructions generated by the guidedinstruction module 225 may be fetched. The preferred medium module 227may extract the user details from the database 103 and based on the userdetails it may use suitable techniques to adapt the messages to be givento the handheld device 101, which outputs the messages in the form ofaudio format, text format or braille script format as requested by theuser.

The assisting module 223 may communicate with the handheld device 101through the user preferred medium to assist the user.

FIG. 3 illustrates a flowchart showing a method for populating adatabase to be used in assisting user with product handling inaccordance with some embodiments of present disclosure.

As illustrated in FIG. 3, the method 300 includes one or more blocks forpopulating the database 103. The method 300 may be described in thegeneral context of computer executable instructions. Generally, computerexecutable instructions can include routines, programs, objects,components, data structures, procedures, modules, and functions, whichperform particular functions or implement particular abstract datatypes.

The order in which the method 300 is described is not intended to beconstrued as a limitation, and any number of the described method blockscan be combined in any order to implement the method. Additionally,individual blocks may be deleted from the methods without departing fromthe scope of the subject matter described herein. Furthermore, themethod can be implemented in any suitable hardware, software, firmware,or combination thereof.

At block 301, the receiver module 215 may receive at least one ofhistoric images of products, historic images of neighbouring products,the historic orientations of products and neighbouring productsassociated with the rack from the IoT sensors. The historic images andhistoric orientations refer to images and orientations received duringtraining phase.

At block 303, the extraction module 213 may extracting the historicproduct details of the products from the historic images of products andthe historic product details of the neighbouring products from thehistoric images of neighbouring products. The historic product detailsmay comprise at least one of product name, product type, product beingproperly oriented or improperly oriented, product handling instruction,the rack information and product characteristics.

At block 305, the receiver module 215 may receive the historic userdetails from the handheld device associated with the user. The historicuser details may comprise at least one of user age, user gender, userpreferences, user behavioural characteristics, user disabilitycharacteristics, user height and user weight. The historic user detailsmay be provided by a user through a handheld device associated with theuser.

At block 307, the other modules 229 may store at least one of thehistoric images of products, the historic images of neighbouringproducts, the historic orientations of products and neighbouringproducts, the historic product details of the products and theneighbouring products and the historic user details in the database 103.

At block 309, the other modules 229 may update at least one of thehistoric images of products, the historic images of neighbouringproducts, the historic orientations of products and neighbouringproducts, the historic product details of the products and theneighbouring products and the historic user details in the database 103for adaptive learning. The update may happen at pre-defined intervals oftime.

FIG. 4a -FIG. 4b illustrates an exemplary representation of assistinguser with product handling in a retail shopping in real-time inaccordance with some embodiments of present disclosure.

As illustrated in FIG. 4a -FIG. 4b , the method 400 includes one or moreblocks for assisting user with product handling in a retail shopping inreal-time. The method 400 may be described in the general context ofcomputer executable instructions. Generally, computer executableinstructions can include routines, programs, objects, components, datastructures, procedures, modules, and functions, which perform particularfunctions or implement particular abstract data types.

The order in which the method 400 is described is not intended to beconstrued as a limitation, and any number of the described method blockscan be combined in any order to implement the method. Additionally,individual blocks may be deleted from the methods without departing fromthe scope of the subject matter described herein. Furthermore, themethod can be implemented in any suitable hardware, software, firmware,or combination thereof.

At block 401, the extraction module 213 may extract a list of productsfrom the handheld device 101 associated with a user and historic userdetails of the user from the database 103. The list of products may beentered or loaded by the user in the handheld device 101.

At block 403, the receiver module 215 may receive a plurality of imageframes of one or more products from the list of products andneighbouring products in a rack from the handheld device 101 and IoTsensors attached to the rack. The image frames may include at least oneof image and video of the one or more products.

At block 405, the identifier module 217 may identify at least one ofproduct type of the one or more products, neighbouring product type ofthe neighbouring products and orientations of the one or more productsand the neighbouring products from the plurality of image frames andrack information corresponding to the rack stored in the database 103.

At block 407, the comparison module 219 may compare the identifiedorientations of the one or more products and the neighbouring productswith historic orientations of products and neighbouring productsassociated with the rack from the database 103.

At block 409, the identifier module 217 may identify at least oneproduct from the one or more products and the neighbouring products thatis improperly oriented based on comparison performed at block 407.

At block 411, the determiner module 221 may determine product safetyhandling instructions of the identified at least one product based onhistoric product details stored in the database 103 and the identifiedorientations of the at least one product at block 409.

At block 413, the assisting module 223 may assist the user in real-timewith handling of the identified at least one product based on theproduct safety handling instructions.

FIG. 4b illustrates a flowchart showing a method for communicatingproduct safety handling instructions into a user preferred medium inaccordance with some embodiments of present disclosure.

At block 415, the guided instruction module 225 may convert the productsafety handling instructions to customized guided instructions based onthe historic user details and the historic product details.

At block 417, the preferred medium module 227 may convert the customizedguided instructions into a user preferred medium based on the historicuser details. The user preferred medium may comprise at least one ofaudio format, video format, text format and braille script format.

At block 419, the assisting module 223 may communicate with the handhelddevice 101 through the user preferred medium to assist the user.

Few examples are presented below based on FIGS. 4a and 4 b.

Example 1: John, an old professor, visits a mall for shopping. Whileexploring the items to purchase in the mall, he reaches grocery section.As John was about to pick a packet of cooking oil, the productassistance system of the present disclosure running on his mobile deviceimmediately alerts him with a text message “Hey, the oil packets aregreasy. Please hold them with a tissue placed on the backside”. Thissaves John from getting his hands dirty. Continuing with his shopping,John then visits kitchen knife section. The product assistance system ofthe present disclosure once again warns him with a new text message“Hey, watch out while pulling the knife. The one close to you has comeout from the wrapper and obliquely placed. Pull it from the rear end ifyou want it”.

Example 2: Tina visits a mall to buy household items. She takes apieceof a glassware and places it back obliquely near the edge of a rack,which is likely to fall. The product assistance system of the presentdisclosure running on her mobile device alerts her through an audiomessage “Hey, request to keep the glassware away from the edge”. Thissaves from the potential destruction of the glassware.

Some of the advantages of the present disclosure are listed below.

The present disclosure reduces risk of user getting injured or harmeddue to dangerous items in a shop left in improper condition orunintended orientation by providing alerts to the user.

The present disclosure provides guidance to the user on how to handle anitem safely, which otherwise could lead to injury to the user or damageto the item.

The present disclosure supports checking of dynamic evolution ororientation of object placement in real time.

Computing System

FIG. 5 illustrates a block diagram of an exemplary computer system 500for implementing embodiments consistent with the present disclosure. Inan embodiment, the computer system 500 may be used to implement theproduct assistance system 107. The computer system 500 may include acentral processing unit (“CPU” or “processor”) 502. The processor 502may include at least one data processor for assisting user with producthandling in a retail shopping in real-time. The processor 502 mayinclude specialized processing units such as, integrated system (bus)controllers, memory management control units, floating point units,graphics processing units, digital signal processing units, etc.

The processor 502 may be disposed in communication with one or moreinput/output (I/O) devices (not shown) via I/O interface 501. The I/Ointerface 501 employ communication protocols/methods such as, withoutlimitation, audio, analog, digital, monoaural, Radio Corporation ofAmerica (RCA) connector, stereo, IEEE-1394 high speed serial bus, serialbus, Universal Serial Bus (USB), infrared, Personal System/2 (PS/2)port, Bayonet Neill-Concelman (BNC) connector, coaxial, component,composite, Digital Visual Interface (DVI), High-Definition MultimediaInterface (HDMI), Radio Frequency (RF) antennas, S-Video, Video GraphicsArray (VGA), IEEE 802.11b/g/n/x, Bluetooth, cellular e.g., Code-DivisionMultiple Access (CDMA), High-Speed Packet Access (HSPA+), Global Systemfor Mobile communications (GSM), Long-Term Evolution (LTE), Worldwideinteroperability for Microwave access (WiMax), or the like, etc.

Using the I/O interface 501, the computer system 500 may communicatewith one or more I/O devices such as input devices 512 and outputdevices 513. For example, the input devices 512 may be an antenna,keyboard, mouse, joystick, (infrared) remote control, camera, cardreader, fax machine, dongle, biometric reader, microphone, touch screen,touchpad, trackball, stylus, scanner, storage device, transceiver, videodevice/source, etc. The output devices 513 may be a printer, faxmachine, video display (e.g., Cathode Ray Tube (CRT), Liquid CrystalDisplay (LCD), Light-Emitting Diode (LED), plasma, Plasma Display Panel(PDP), Organic Light-Emitting Diode display (OLED) or the like), audiospeaker, etc.

In some embodiments, the computer system 500 consists of the productassistance system 107. The processor 502 may be disposed incommunication with the communication network 509 via a network interface503. The network interface 503 may communicate with the communicationnetwork 509. The network interface 503 may employ connection protocolsincluding, without limitation, direct connect, Ethernet (e.g., twistedpair 10/100/1000 Base T), Transmission Control Protocol/InternetProtocol (TCP/IP), token ring, IEEE 802.11a/b/g/n/x, etc. Thecommunication network 509 may include, without limitation, a directinterconnection, Local Area Network (LAN), Wide Area Network (WAN),wireless network (e.g., using Wireless Application Protocol), theInternet, etc. Using the network interface 503 and the communicationnetwork 509, the computer system 500 may communicate with a database514. The network interface 503 may employ connection protocols include,but not limited to, direct connect, Ethernet (e.g., twisted pair10/100/1000 Base T), Transmission Control Protocol/Internet Protocol(TCP/IP), token ring, IEEE 802.11a/b/g/n/x, etc.

The communication network 509 includes, but is not limited to, a directinterconnection, a Peer to Peer (P2P) network, Local Area Network (LAN),Wide Area Network (WAN), wireless network (e.g., using WirelessApplication Protocol), the Internet, Wi-Fi and such. The first networkand the second network may either be a dedicated network or a sharednetwork, which represents an association of the different types ofnetworks that use a variety of protocols, for example, HypertextTransfer Protocol (HTTP), Transmission Control Protocol/InternetProtocol (TCP/IP), Wireless Application Protocol (WAP), etc., tocommunicate with each other. Further, the first network and the secondnetwork may include a variety of network devices, including routers,bridges, servers, computing devices, storage devices, etc.

In some embodiments, the processor 502 may be disposed in communicationwith a memory 505 (e.g., RAM, ROM, etc. not shown in FIG. 5) via astorage interface 504. The storage interface 504 may connect to memory505 including, without limitation, memory drives, removable disc drives,etc., employing connection protocols such as, Serial Advanced TechnologyAttachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394,Universal Serial Bus (USB), fiber channel, Small Computer SystemsInterface (SCSI), etc. The memory drives may further include a drum,magnetic disc drive, magneto-optical drive, optical drive, RedundantArray of Independent Discs (RAID), solid-state memory devices,solid-state drives, etc.

The memory 505 may store a collection of program or database components,including, without limitation, user interface 506, an operating system507, etc. In some embodiments, computer system 500 may storeuser/application data, such as, the data, variables, records, etc., asdescribed in this disclosure. Such databases may be implemented asfault-tolerant, relational, scalable, secure databases such as Oracle orSybase.

The operating system 507 may facilitate resource management andoperation of the computer system 500. Examples of operating systemsinclude, without limitation, APPLE® MACINTOSH® OS X®, UNIX®, UNIX-likesystem distributions (E.G., BERKELEY SOFTWARE DISTRIBUTION® (BSD),FREEBSD®, NETBSD®, OPENBSD, etc.), LINUX® DISTRIBUTIONS (E.G., RED HAT®,UBUNTU®, KUBUNTU®, etc.), IBM®OS/2®, MICROSOFT® WINDOWS® (XP®,VISTA®/7/8, 10 etc.), APPLE® IOS®, GOOGLE™ ANDROID™, BLACKBERRY® OS, orthe like.

In some embodiments, the computer system 500 may implement web browser508 stored program components. Web browser 508 may be a hypertextviewing application, such as MICROSOFT® INTERNET EXPLORER®, GOOGLE™CHROME™, MOZILLA® FIREFOX®, APPLE® SAFARI®, etc. Secure web browsing maybe provided using Secure Hypertext Transport Protocol (HTTPS), SecureSockets Layer (SSL), Transport Layer Security (TLS), etc. Web browsers508 may utilize facilities such as AJAX, DHTML, ADOBE® FLASH®,JAVASCRIPT®, JAVA®, Application Programming Interfaces (APIs), etc. Thecomputer system 500 may implement a mail server (not shown in FIG. 5)stored program component. The mail server may be an Internet mail serversuch as Microsoft Exchange, or the like. The mail server may utilizefacilities such as ASP, ACTIVEX®, ANSI® C++/C #, MICROSOFT®, .NET, CGISCRIPTS, JAVA®, JAVASCRIP®, PERL®, PHP, PYTHON®, WEBOBJECTS®, etc. Themail server may utilize communication protocols such as Internet MessageAccess Protocol (IMAP), Messaging Application Programming Interface(MAPI), MICROSOFT® exchange, Post Office Protocol (POP), Simple MailTransfer Protocol (SMTP), or the like. The computer system 500 mayimplement a mail client (not shown in FIG. 5) stored program component.The mail client may be a mail viewing application, such as APPLE® MAIL,MICROSOFT® ENTOURAGE®, MICROSOFT® OUTLOOK®, MOZILLA® THUNDERBIRD®, etc.

Furthermore, one or more computer-readable storage media may be utilizedin implementing embodiments consistent with the present disclosure. Acomputer-readable storage medium refers to any type of physical memoryon which information or data readable by a processor may be stored.Thus, a computer-readable storage medium may store instructions forexecution by one or more processors, including instructions for causingthe processor(s) to perform steps or stages consistent with theembodiments described herein. The term “computer-readable medium” shouldbe understood to include tangible items and exclude carrier waves andtransient signals, i.e., be non-transitory. Examples include RandomAccess Memory (RAM), Read-Only Memory (ROM), volatile memory,non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks,and any other known physical storage media.

The described operations may be implemented as a method, system orarticle of manufacture using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof. The described operations may be implemented as code maintainedin a “non-transitory computer readable medium”, where a processor mayread and execute the code from the computer readable medium. Theprocessor is at least one of a microprocessor and a processor capable ofprocessing and executing the queries. A non-transitory computer readablemedium may include media such as magnetic storage medium (e.g., harddisk drives, floppy disks, tape, etc.), optical storage (CD-ROMs, DVDs,optical disks, etc.), volatile and non-volatile memory devices (e.g.,EEPROMs, ROMs, PROMs, RAMs, DRAMs, SRAMs, Flash Memory, firmware,programmable logic, etc.), etc. Further, non-transitorycomputer-readable media include all computer-readable media except for atransitory. The code implementing the described operations may furtherbe implemented in hardware logic (e.g., an integrated circuit chip,Programmable Gate Array (PGA), Application Specific Integrated Circuit(ASIC), etc.).

The terms “an embodiment”, “embodiment”, “embodiments”, “theembodiment”, “the embodiments”, “one or more embodiments”, “someembodiments”, and “one embodiment” mean “one or more (but not all)embodiments of the invention(s)” unless expressly specified otherwise.

The terms “including”, “comprising”, “having” and variations thereofmean “including but not limited to”, unless expressly specifiedotherwise.

The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise.

The terms “a”, “an” and “the” mean “one or more”, unless expresslyspecified otherwise.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Onthe contrary, a variety of optional components are described toillustrate the wide variety of possible embodiments of the invention.

When a single device or article is described herein, it will be readilyapparent that more than one device/article (whether or not theycooperate) may be used in place of a single device/article. Similarly,where more than one device or article is described herein (whether ornot they cooperate), it will be readily apparent that a singledevice/article may be used in place of the more than one device orarticle or a different number of devices/articles may be used instead ofthe shown number of devices or programs. The functionality and/or thefeatures of a device may be alternatively embodied by one or more otherdevices which are not explicitly described as having suchfunctionality/features. Thus, other embodiments of the invention neednot include the device itself.

The illustrated operations of FIGS. 3, 4 a and 4 b show certain eventsoccurring in a certain order. In alternative embodiments, certainoperations may be performed in a different order, modified or removed.Moreover, steps may be added to the above described logic and stillconform to the described embodiments. Further, operations describedherein may occur sequentially or certain operations may be processed inparallel. Yet further, operations may be performed by a singleprocessing unit or by distributed processing units.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based here on. Accordingly, the disclosure of theembodiments of the invention is intended to be illustrative, but notlimiting, of the scope of the invention, which is set forth in thefollowing claims.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

REFERRAL NUMERALS: Reference number Description 100 Environment 101Handheld device 103 Database 105 Communication network 107 Productassistance system 111 I/O interface 113 Memory 115 Processor 200 Data201 Product list data 203 Image frame data 205 Other data 211 Modules213 Extraction module 215 Receiver module 217 Identifier module 219Comparison module 221 Determiner module 223 Assisting module 225 Guidedinstruction module 227 Preferred medium module 229 Other modules 500Computer system 501 I/O interface 502 Processor 503 Network interface504 Storage interface 505 Memory 506 User interface 507 Operating system508 Web browser 509 Communication network 512 Input devices 513 Outputdevices 514 Database

What is claimed is:
 1. A method of assisting user with product handlingin a retail shopping in real-time, the method comprising: extracting, bya product assistance system, a list of products from a handheld deviceassociated with a user and historic user details of the user from adatabase; receiving, by the product assistance system, a plurality ofimage frames of one or more products from the list of products andneighbouring products in a rack from the handheld device and Internet ofThings (IoT) sensors attached to the rack; identifying, by the productassistance system, at least one of product type of the one or moreproducts, neighbouring product type of the neighbouring products andorientations of the one or more products and the neighbouring productsfrom the plurality of image frames and rack information corresponding tothe rack stored in the database; comparing, by the product assistancesystem, the identified orientations of the one or more products and theneighbouring products with historic orientations of products andneighbouring products associated with the rack from the database;identifying, by the product assistance system, using an artificialneural network, at least one product from the one or more products andthe neighbouring products that is improperly oriented based oncomparison; determining, by the product assistance system, using theartificial neural network, product safety handling instructions of theidentified at least one product based on historic product details storedin the database and the identified orientations of the at least oneproduct; receiving, by the product assistance system, at least one ofhistoric images of products, historic images of neighbouring products,the historic orientations of products and neighbouring productsassociated with the rack from the IoT sensors; extracting, by theproduct assistance system, the historic product details of the productsfrom the historic images of products and the historic product details ofthe neighbouring products from the historic images of neighbouringproducts; receiving, by the product assistance system, the historic userdetails from the handheld device associated with the user; storing, bythe product assistance system, at least one of the historic images ofproducts, the historic images of neighbouring products, the historicorientations of products and neighbouring products, the historic productdetails of the products and the neighbouring products and the historicuser details in the database; and assisting, by the product assistancesystem, the user in real-time with handling of the identified at leastone product based on the product safety handling instructions.
 2. Themethod as claimed in claim 1, wherein assisting the user in real-timecomprising: converting, by the product assistance system, the productsafety handling instructions to customized guided instructions based onthe historic user details and the historic product details; converting,by the product assistance system, the customized guided instructionsinto a user preferred medium based on the historic user details; andcommunicating, by the product assistance system, with the handhelddevice through the user preferred medium to assist the user.
 3. Themethod as claimed in claim 1, the method further comprising: providing,by the product assistance system, a shopping route to access the list ofproducts in a shop based on at least one of the list of products, thehistoric user details and product purchase priority of the user.
 4. Themethod as claimed in claim 1, the method further comprising: updating,by the product assistance system, at least one of the historic images ofproducts, the historic images of neighbouring products, the historicorientations of products and neighbouring products, the historic productdetails of the products and the neighbouring products and the historicuser details in the database for adaptive learning.
 5. The method asclaimed in claim 1, wherein the user preferred medium comprises at leastone of audio format, video format, text format and braille scriptformat.
 6. The method as claimed in claim 1, wherein the historic userdetails comprise at least one of user age, user gender, userpreferences, user behavioural characteristics, user disabilitycharacteristics, user height and user weight.
 7. The method as claimedin claim 1, wherein the historic product details comprise at least oneof product name, product type, product being properly oriented orimproperly oriented, product handling instruction, the rack informationand product characteristics.
 8. A product assistance system forassisting user with product handling in a retail shopping in real-time,the product assistance system comprising: a processor; and a memorycommunicatively coupled to the processor, wherein the memory storesprocessor-executable instructions, which on execution, cause theprocessor to: extract a list of products from a handheld deviceassociated with a user and historic user details of the user from adatabase; receive a plurality of image frames of one or more productsfrom the list of products and neighbouring products in a rack from thehandheld device and Internet of Things (IoT) sensors attached to therack; identify at least one of product type of the one or more products,neighbouring product type of the neighbouring products and orientationsof the one or more products and the neighbouring products from theplurality of image frames and rack information corresponding to the rackstored in the database; compare the identified orientations of the oneor more products and the neighbouring products with historicorientations of products and neighbouring products associated with therack from the database; identify, using an artificial neural network, atleast one product from the one or more products and the neighbouringproducts that is improperly oriented based on comparison; determine,using the artificial neural network, product safety handlinginstructions of the identified at least one product based on historicproduct details stored in the database and the identified orientationsof the at least one product; receive at least one of historic images ofproducts, historic images of neighbouring products and the historicorientations of products and neighbouring products associated with therack from the IoT sensors; extract the historic product details of theproducts from the historic images of products and the historic productdetails of the neighbouring products from the historic images ofneighbouring products; receive the historic user details from thehandheld device associated with the user; store at least one of thehistoric images of products, the historic images of neighbouringproducts, the historic orientations of products and neighbouringproducts, the historic product details of the products and theneighbouring products and the historic user details in the database; andassist the user in real-time with handling of the identified at leastone product based on the product safety handling instructions.
 9. Theproduct assistance system as claimed in claim 8, the product assistancesystem causes the processor to: convert the product safety handlinginstructions to customized guided instructions based on the historicuser details and the historic product details; convert the customizedguided instructions into a user preferred medium based on the historicuser details; and communicate with the handheld device through the userpreferred medium to assist the user.
 10. The product assistance systemas claimed in claim 9, the product assistance system causes theprocessor to: provide a shopping route to access the list of products ina shop based on at least one of the list of products, the historic userdetails and product purchase priority of the user.
 11. The productassistance system as claimed in 8, the product assistance system causesthe processor to: update at least one of the historic images ofproducts, the historic images of neighbouring products, the historicorientations of products and neighbouring products, the historic productdetails of the products and the neighbouring products and the historicuser details in the database for adaptive learning.
 12. The productassistance system as claimed in 8, wherein the user preferred mediumcomprises at least one of audio format, video format, text format andbraille script format.
 13. The product assistance system as claimed in8, wherein the historic user details comprise at least one of user age,user gender, user preferences, user behavioural characteristics, userdisability characteristics, user height and user weight.
 14. The productassistance system as claimed in 8, wherein the historic product detailscomprise at least one of product name, product type, product beingproperly oriented or improperly oriented, product handling instruction,the rack information and product characteristics.
 15. A non-transitorycomputer readable medium including instructions stored thereon that whenprocessed by at least one processor cause a product assistance system toperform operations comprising: extracting a list of products from ahandheld device associated with a user and historic user details of theuser from a database; receiving a plurality of image frames of one ormore products from the list of products and neighbouring products in arack from the handheld device and Internet of Things (IoT) sensorsattached to the rack; identifying at least one of product type of theone or more products, neighbouring product type of the neighbouringproducts and orientations of the one or more products and theneighbouring products from the plurality of image frames and rackinformation corresponding to the rack stored in the database; comparingthe identified orientations of the one or more products and theneighbouring products with historic orientations of products andneighbouring products associated with the rack from the database;identifying, using an artificial neural network, at least one productfrom the one or more products and the neighbouring products that isimproperly oriented based on comparison; determining, using theartificial neural network, product safety handling instructions of theidentified at least one product based on historic product details storedin the database and the identified orientations of the at least oneproduct; receiving at least one of historic images of products, historicimages of neighbouring products and the historic orientations ofproducts and neighbouring products associated with the rack from the IoTsensors; extracting the historic product details of the products fromthe historic images of products and the historic product details of theneighbouring products from the historic images of neighbouring products;receiving the historic user details from the handheld device associatedwith the user; and storing at least one of the historic images ofproducts, the historic images of neighbouring products, the historicorientations of products and neighbouring products, the historic productdetails of the products and the neighbouring products and the historicuser details in the database; and assisting the user in real-time withhandling of the identified at least one product based on the productsafety handling instructions.
 16. The non-transitory computer readablemedium as claimed in claim 15, wherein the instructions cause theprocessor to: convert the product safety handling instructions tocustomized guided instructions based on the historic user details andthe historic product details; convert the customized guided instructionsinto a user preferred medium based on the historic user details; andcommunicate with the handheld device through the user preferred mediumto assist the user.
 17. The non-transitory computer readable medium asclaimed in claim 15, wherein the instructions cause the processor to:provide a shopping route to access the list of products in a shop basedon at least one of the list of products, the historic user details andproduct purchase priority of the user.